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Introduction: The Ottawa Knee Rule is a validated clinical decision rule for determining whether 
knee radiographs should be obtained in the setting of acute knee trauma. The objectives of this 
study were to assess physician knowledge of, barriers to implementation of, and compliance with 
the Ottawa Knee Rule in academic emergency departments (EDs), and evaluate whether patient 
characteristics predict guideline noncompliance. 

Methods: A 10 question online survey was distributed to all attending ED physicians working at three 
affiliated academic EDs to assess knowledge, attitudes and self-reported practice behaviors related 
to the Ottawa Knee Rule. We also performed a retrospective ED record review of patients 13 years 
of age and older who presented with acute knee trauma to the 3 study EDs during the 2009 calendar 
year, and we analyzed ED records for 19 variables. 

Results: ED physicians (n = 47) correctly answered 73.2% of questions assessing knowledge of the 
Ottawa Knee Rule. The most commonly cited barriers to implementation were "patient expectations" 
and system issues, such as "orthopedics referral requirement." We retrospectively reviewed 838 
records, with 260 eligible for study inclusion. The rate of Ottawa Knee Rule compliance was 
retrospectively determined to be 63.1%. We observed a statistically significant correlation between 
Ottawa Knee Rule compliance and patient age, but not gender, insurance status, or provider type, 
among others. 

Conclusion: Compliance with the Ottawa Knee Rule among academic ED healthcare providers is 
poor, which was predicted by patient age and not other physician or patient variables. Improving 
compliance will require comprehensive educational and systemic interventions. [West J Emerg Med. 
2012;13(4):366-373.] 



INTRODUCTION 

Knee pain is a common presenting complaint 
in the emergency department (ED) and accounts for 
approximately 1.3 million ED visits annually in the 
United States (US). 1 Despite the prevalence of acute knee 
trauma, fractures are only observed in approximately 
6% of patients. 2 Knee radiographs to detect fractures, 
however, remain one of the most commonly ordered 
studies and account for $1 billion in healthcare spending 
annually. 3 - 4 



The Ottawa Knee Rule was published in 1995 by Stiell et 
al 5 as a tool for determining whether knee radiographs should 
be obtained to detect a fracture in the setting of acute knee 
trauma. The rule states that knee radiographs are indicated 
in these patients if at least one of the following criteria is 
satisfied: the patient is at least 55 years old, has an inability to 
bear weight immediately after trauma and in the ED for four 
steps (regardless of limp), has isolated patellar tenderness, 
fibular head tenderness, or an inability to flex the knee to 
90<> i,5,6 stiell et al 1 validated the rule in 2 prospective studies, 
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which showed the criteria to be 1 00% sensitive for identifying 
clinically-significant fractures. Additionally, Bachmann et 
al 7 published a meta-analysis concluding that the Ottawa 
Knee Rule was 99% sensitive and 49% specific for knee 
fracture. While the criteria were initially validated in adults, 
prospective studies and meta-analyses also demonstrated that 
they are 99-100% sensitive and 43-46% specific in children 
over 5 years of age. 8 - 9 

When used appropriately the Ottawa Knee Rule has been 
shown to reduce unnecessary imaging. In an implementation 
trial conducted by Stiell et al 6 , educational interventions on 
the Ottawa Knee Rule targeting ED physicians decreased the 
proportion of knee radiographs by 26.4%, ED visit costs by 
$34 per patient, and time spent in the ED by 33.1 minutes. 10 
Another prospective trial demonstrated a 35% decrease in 
knee radiographs. 11 

Despite evidence supporting use of the Ottawa Knee Rule, 
its adoption has been limited. A multi-national survey of ED 
physicians 10 years ago investigated this issue. 12 According 
to the results, self-reported knowledge of the Ottawa Knee 
Rule was highest in English-speaking countries, yet use of 
the rule among those with knowledge of the criteria was 
lowest in the U.S. While the majority of respondents agreed 
that clinical decision rules are intended to reduce healthcare 
costs and improve quality of care, physicians in the U.S. 
were most likely to agree that such rules are oversimplified, 
increase the likelihood of being sued, and challenge physician 
authority. These studies suggest that a multitude of factors 
contribute to limited use of the Ottawa Knee Rule and 
consequent radiograph overuse. Knee radiographs remain 
one of the most commonly performed imaging studies and, 
despite low fracture rates, are obtained in 60-80% of acute 
knee trauma cases. 31314 While the low adoption rate of the 
Ottawa Knee Rule is known, there are currently no published 
studies assessing physician knowledge of the rule, or patient 
or physician predictors of compliance with it. 

In the present study, 1 5 years after publication of this 
clinical decision rule, we aimed to (1) evaluate knowledge, 
attitudes and self-reported practice behaviors regarding 
the Ottawa Knee Rule among attending academic U.S. ED 
physicians; (2) determine Ottawa Knee Rule adherence among 
ED providers; and (3) examine if patient level characteristics 
predict guideline noncompliance. We hypothesized that 
Ottawa Knee Rule adherence is poor. Additionally, we 
hypothesized that systems-based barriers inherent to the ED 
setting and patient factors prevent appropriate application of 
the rule. 

METHODS 

The Institutional Review Boards of the participating hospitals 
and the university approved all aspects of the study protocol. 

ED Physician Survey 

To assess ED physician knowledge, and barriers to 



implementation, of the Ottawa Knee Rule, the authors 
designed a 1 0-question online survey. The questions were 
developed by extrapolating scenarios from the rule criteria, 
and by phrasing inquiries about basic demographic and other 
information in plain language agreed upon by all authors. The 
survey was administered to all 76 ED attending physicians 
(all board certified in emergency medicine and/or pediatric 
emergency medicine) who were faculty at one of the nation's 
largest academic emergency medicine departments in June 
2010. The three study institutions' EDs were staffed by 
members of the same university-based physician group and 
included an academic trauma center, a community teaching 
hospital and an academic pediatric specialty hospital with a 
combined annual volume of approximately 200,000 ED visits. 

Of the questions, 2 related to the demographics of the 
respondents, 5 evaluated knowledge of the Ottawa Knee Rule 
through case vignettes and guideline questions, one probed 
for self-reported adherence to the rule, and 2 inquired about 
potential barriers to implementation (see Appendix for full 
survey questions). One of the barriers-to-implementation 
questions asked respondents to identify the top three reasons 
(out of nine choices) for ordering a knee radiograph in the 
absence of the Ottawa Knee Rule criteria being met; many 
of these answer choices were adapted from Graham et al's 12 
physician survey. We disseminated the survey link via email. 
All data were collected with identification of the respondent, 
and a $5 gift card was offered for survey completion. 
The survey responses, however, were not linked to the 
respondent's name. 

We evaluated knowledge of the Ottawa Knee Rule by 
comparing the proportion of participants who answered the 
5 vignettes and guideline questions correctly with the total 
number of participants. Similarly, we calculated simple 
percentages for the responses to the 2 survey questions 
relating to the most commonly reported barriers. 

Medical Record Review 

To corroborate self-reported data on adherence to 
the Ottawa Knee Rule, as well as collect patient level 
characteristics that may influence guideline compliance, we 
conducted a retrospective ED medical record review. The 
review notes were not directly linked to the physicians who 
responded to the survey. 

Our study population consisted of patients 13 years of 
age and older who presented with acute knee trauma to the 
three study institutions' EDs between January 1, 2009 and 
December 31, 2009. To include the entire teenage population 
and broaden the potential data set we chose 13 as a lower age 
limit Patients were initially identified by querying the ED 
billing database for 16 International Classification of Diseases 
9 (ICD-9) codes related to acute knee and lower extremity 
trauma (716.1, 717, 718.86, 822, 823, 823.1, 823.8, 823.9, 
827, 836, 844, 891, 916, 924, 928, 959.7). We then applied 
the same exclusion criteria used in the Ottawa Knee Rule's 
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validation studies (except for age less than 1 8 years), which 
excluded patients who were referred from outside hospitals 
with knee radiographs, sustained knee trauma more than seven 
days previously, returned for reassessment of the same injury, 
had isolated skin injury, had multiple trauma, were pregnant, 
were paraplegic or had an altered level of consciousness. 1,5 
Two data abstracters then examined medical records of 
eligible patients for 1 9 principal variables gathered from ED 
nursing records, ED physician records and radiology reports. 
We entered data into Microsoft Excel; any missing data were 
noted. The authors managed any discrepancies between 
nursing and physician records by deferring to physician 
documentation for assessment of principal variables. 

We organized the principal variables by categories such 
as "patient characteristics" (e.g. age, gender, insurance status, 
previous knee injury), "injury features" (e.g. mechanism, 
setting, the 5 Ottawa Knee Rule criteria, diagnosis), 
"radiograph ordering" (e.g. imaging, type of provider ordering 
films), and "other" (e.g. time spent in the ED, returned to 
the ED within 2 weeks for reassessment). From these, we 
evaluated Ottawa Knee Rule compliance by comparing the 
proportion of patients who had knee radiographs obtained 
with the proportion of patients in whom knee radiographs 
were indicated according to the rule. We assessed potential 
association of Ottawa Knee Rule adherence with patient 
variables, adjusted for multiple comparisons, using the 
Fisher's Exact test, which was calculated in Statistical 
Analysis System 8.2 (SAS, SAS Institute Inc., Cary, NC). 
Calculations involving time spent in the ED excluded patients 
who were admitted to the hospital or taken to the operating 
room, and were made using the Wilcoxon Rank Sum Test. We 
determined inter-rater reliability between the 2 reviewers by 
calculating the Cohen's kappa value based upon whether the 
Ottawa Knee Rule criteria were met from a random subset of 
33 medical records. 

RESULTS 

ED Physician Survey 

Forty-seven out of 76 ED physicians responded to our 
survey (61.8% response rate). On average, respondents 
worked 22.3 (standard deviation (SD) 7.8, 95% confidence 
interval (CI) 20.1-24.5) hours per week in the ED and had 8.7 
(SD 8.3, CI 6.4-11.1) years of experience. 

Physicians scored an average of 73.2% (CI 66.6-79.8) 
on questions assessing Ottawa Knee Rule knowledge, with 
2.1% of respondents correctly answering all of the questions, 
78.7% answering all but 1 of the questions correctly, and 0% 
answering all of the questions incorrectly. Only 36.2% of 
responding physicians, however, chose to withhold diagnostic 
imaging in a case vignette that did not satisfy the Ottawa Knee 
Rule criteria (an ambulatory adolescent with a severe limp 
who exhibited full range of motion of the knee with no focal 
tenderness). 

Generally, self-reported adherence to the Ottawa Knee 



Rule was poor. More than one -third (36.2%) of physicians 
reported never using the guideline, while only 23.4% of 
physicians used the rule "always" or "most of the time." The 
most commonly cited barriers to rule implementation were 
"patient expectations" and "patient satisfaction" (Figure). 
Other barriers frequently identified included an orthopedics 
referral requirement and lack of confidence in physical exam 
findings. Finally, ED physicians reported that the majority 
(53.2%) of radiographs for knee injury patients were ordered 
by non-attending physician providers (e.g. residents, triage 
nurses and physician assistants). 

Medical Record Review 

The ED billing database query for ICD-9 codes, restricted 
by target ages and dates, identified 838 patient visits. Upon 
review of these records, 437 did not have knee trauma, 
129 met exclusion criteria, and 12 did not record sufficient 
information to determine if the Ottawa Knee Rule criteria 
were met. Consequently, 260 records were eligible for study 
inclusion. The inter-rater reliability was high (Cohen's kappa 
= 0.939).The demographics, community setting, and etiology 
of acute knee trauma in our study population are summarized 
in Table 1. Of the 260 patients, 198 (76.2%) had a knee 
radiograph and 1 had a magnetic resonance image (Table 
2). Forty-one patients had clinically-significant fractures (39 
involving the patella). Only 17 patients returned to the ED 
within 2 weeks for reevaluation of the same injury. Of these 
patients, 1 6 had radiographs at the initial visit and none had 
a revised diagnosis based on reevaluation. The mean time 
spent in the ED was 4.1 hours (median 3.4; interquartile 
range 2.7 hours). Upon excluding the 12 patients who were 
admitted or taken to the operating room, the mean time was 
3.9 hours (median 3.3; interquartile range 2.3 hours). Patients 
who had a radiograph spent more time in the ED (mean 4. 1, 
median 3.5, interquartile range 2.5 hours) than those who did 
not (mean 3.2, median 2.8, interquartile range 1.6 hours) (p 
< 0.05). Ottawa Knee Rule criteria fulfillment is summarized 
in Table 3. While some data were missing from the medical 
records, all cases had sufficient information to assess rule 




Figure. Self-reported barriers to implementation of the Ottawa Knee 
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Table 1. Acute knee trauma patient demographics and injury 



description. 

Characteristic Value 
Age (years) 

Mean (Median) 43.3(41.5) 

Interquartile Range 39 
Gender (%) 

Male 51.5 

Female 48.5 

Insurance (%) 

Yes 80.4 

No 19.8 

Previous ipsilateral knee injury (%) 7.3 

Injury setting (%) 

Home 41.9 

Street 24.6 

Sports 13.8 

Other 10.0 

Mechanism of injury* (%) 

Fall 60.4 

Twisting 14.6 

Direct blow 11.2 
*Patient may have more than one mechanism of injury. 

Table 2. Acute knee trauma diagnosis. 

Characteristic Value 

Radiography performed (%) 76.2 

Provider ordering film (%) 

Attending physician 32.3 

Resident 21.7 

Physician assistant 17.7 

Nurse 22.7 

Undetermined 5.6 

Diagnosis* (%) 

Abrasion 38.8 

Fracture 15.8 

Ligament injury 14.6 

Meniscus injury 10.4 

Contusion 9.6 

Time spent in ED (hours) 

Mean (Median) 4.1 (3.4) 

Interquartile Range 2.7 

Returned within 2 weeks (%) 6.5 
*Patient may have more than one diagnosis. 



fulfillment. Taken together, 65.4% of patients fulfilled 
at least 1 of the Ottawa Knee Rule criteria and therefore 
warranted a knee radiograph. Additionally, the rule was 1 00% 
sensitive and 40.7% specific for fracture in the 260 cases. 
We assessed Ottawa Knee Rule compliance by comparing 
criteria fulfillment with radiograph obtainment (Table 4). Out 
of 260 cases, 1 64 had knee radiographs either appropriately 
obtained or appropriately not obtained, and were therefore 
considered compliant with the Ottawa Knee Rule. Conversely, 
96 cases had knee radiographs either inappropriately obtained 
or inappropriately not obtained, and were considered 
noncompliant. Patients who had a radiograph inappropriately 
obtained spent an average of 3.7 hours (median 3.5; 
interquartile range 1.7 hours) in the ED. Overall, the rate of 
Ottawa Knee Rule compliance was 63.1%. 

The association between Ottawa Knee Rule compliance 
and various patient and provider factors is summarized in 
Table 5. A statistically significant association (p = 0.01) was 
only observed with patient age. Compliance was higher 
inpatients aged 13-18 years old (76.5%) and>55 years 
old (68.7%), as compared to 19-54 year olds (54.0%). No 
statistically significant (p<0.05) differences were noted 
with provider type, patient gender, patient insurance status, 
previous ipsilateral knee injury, sports injury or mechanism of 
injury. 

DISCUSSION 

To our knowledge, this is the first retrospective record 
review to analyze Ottawa Knee Rule compliance and potential 
associations between compliance with provider and patient 
factors. 

The survey response rate was acceptable, and consistent 
with other surveys targeting physicians. 15 The survey 
demonstrated that ED physician knowledge of the Ottawa 
Knee Rule was good, but self-reported adherence was poor. 
Interestingly, of the 5 vignettes, the scenario with the lowest 
correct response rate (36.2%) was the only case in which 
imaging was not indicated according to the Ottawa Knee 
Rule. That is, despite overall acceptable knowledge of the 
Ottawa Knee Rule criteria and its application, physicians 
were still hesitant to withhold imaging. This was also found 
in our record review, where one-third of radiographs were 
ordered for patients not meeting any criteria. Furthermore, 
physicians noted that the primary barriers to Ottawa Knee 
Rule implementation were related to patient and systems 
barriers rather than the criteria themselves. The results of our 
recent survey coupled with the early findings by Graham et 
al 12 suggest that noncompliance with the Ottawa Knee Rule 
is currently likely more attributable to systemic concerns, 
such as orthopedic consultation demands and malpractice 
implications (as indicated by the "Legal concern" column in 
Figure), than lack of knowledge. Addressing these systemic 
concerns is important to maximize adherence to the Ottawa 
Knee Rule. 
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Table 3. Ottawa Knee Rule criteria fulfillment. 



Criteria 


Yes (%) 


No (%) 


Undetermined 1 " (%) 


Age > 55 years 


83 (31.9) 


177 (68.1) 


0(0) 


Isolated patellar 
tenderness 


47 (18.1) 


173 (66.5) 


40 (15.4) 


Fibular head 
tenderness 


4(1.5) 


198 (76.2) 


58 (22.3) 


Inability to flex 
knee to 90° 


88 (34.2) 


157 (60.4) 


15(5.4) 


Non-weight 


17 (6.5) 


88 (33.8) 


155 (59.6) 



Table 5. Association of Ottawa Knee Rule compliance with patient 
and provider factors. 



bearing after 
injury and in ED 



Criteria met 
(overall)* 



170(65.4) 90 (34.6) 



0(0) 



f Refers to cases where limited or missing data precluded an 
assessment. 

* Indicates cases that met at least one of the Ottawa Knee Rule 
criteria, thus warranting a knee radiograph. 



Table 4. Ottawa Knee Rule compliance as determined by ED 
medical record review. 



Ottawa Knee 


Knee Radiograph 


Rule Criteria 


Performed 


Not Performed 


Met 


136 


35 


Not Met 


61 


28 



Poor compliance with the Ottawa Knee Rule, as 
reported by ED physicians in our survey, was confirmed by 
retrospective review. This demonstrated a compliance rate of 
63.1%. Overall, 76.2% of patients received knee radiographs, 
consistent with the previously published result of 74% in Stiell 
et al's 3 retrospective study. Our study population, however, 
had a fracture rate of 15.8%, which is higher than previously 
published figures of 6-7%. 2 - 3 This discrepancy may be due 
to the fact that 2 of our study hospitals are Level I trauma 
centers. 

Additionally, physician respondents to our survey reported 
that the majority of radiographs were ordered by non-attending 
physician providers. Review of medical records confirmed 
this - only 34.4% of radiographs were ordered by attending 
physicians, while 24.2% were ordered by residents, 18.8% 
by nurses, and 22.6% by physician assistants. Recognizing 
that other healthcare providers influence radiograph ordering, 
Matteucci et al 16 performed a prospective study in which 
both physicians and triage nurses were educated on the 
Ottawa Knee Rule. This training led to 37% and 21% relative 
reductions in radiograph ordering among physicians and triage 
nurses, respectively, although triage nurses still ordered 3.6 
times more radiographs than physicians. Our hypothesis that 



Characteristic 



Compliance 

(%) 



N 



P (Fisher's 
Exact) 



nuviuci iy|jc 






n 71 

u. / o 


Attending physician 


UJ.D 


04 




Resident 


61.9 


45 




Nurse 


75.6 


35 




Physician assistant 


71.4 


42 




Age 






0.01* 


0-18 years old 


76.5 


51 




9-54 years old 


54.0 


126 




>55 years old 


68.7 


83 




Gender 






0.80 


Male 


61.9 


134 




Female 


64.0 


126 




Insurance 






0.75 


Yes 


62.5 


208 




No 


65.4 


52 




Previous knee injury 






0.15 


Yes 


47.4 


19 




No 


64.3 


241 




Sports-related injury 






0.09 


Yes 


75.0 


36 




No 


61.9 


224 




Mechanism of injury** 








Fall 


63.7 


157 


0.90 


Twisting 


65.8 


38 


0.86 


Direct blow 


65.5 


29 


0.84 



* Provider type was undetermined in 12 cases, which have been 

excluded from above analysis. 

f Statistically significant values (p < 0.05). 

** Patient may have more than one mechanism of injury. Thus, 

independent p values were calculated. 

rule compliance would be higher when physicians ordered 
radiographs proved incorrect, as there was no association 
with provider type. Future educational efforts should target all 
ED healthcare providers, as well as consulting and follow-up 
services such as orthopedics, given the significant proportion 
of radiographs ordered by non-attending physicians. 

We further aimed to determine which patient level 
variables correlated with Ottawa Knee Rule compliance. 
Patient age was the only factor to have a statistically 
significant correlation. Compliance was significantly higher 
in younger (< 18 years old) and older (> 55 years old) 
patients, as compared to patients aged 1 9-54 years. The higher 
compliance rate in the older group is consistent with the fact 
that all patients in this age range warrant a radiograph (per 
rule criteria). Additionally, the higher compliance rate in the 
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younger group may be attributed to provider hesitation to 
order imaging in pediatric patients given concerns of radiation 
exposure. 

We also examined whether Ottawa Knee Rule compliance 
correlated with reduced ED wait times and radiograph 
ordering. After excluding patients taken to the operating room 
or admitted to the hospital from the ED, patients who had a 
knee radiograph spent 53 minutes longer in the ED (mean 
of 4. 1 hours) compared to those who received no radiograph 
(mean of 3.2 hours). Since this difference was potentially 
confounded by injury severity, we also performed the analysis 
after excluding fracture diagnoses. Even after removing these 
patients, those who had a knee radiograph spent 47 minutes 
longer in the ED. These figures are comparable to 2 previous 
studies by Stiell et al, 1,6 which showed that patients receiving 
knee radiographs spend 33-39 minutes longer in the ED. 

Finally, the Ottawa Knee Rule was 1 00% sensitive 
and 40.7% specific for fracture in our retrospective study. 
Our study was not implicitly designed to determine these 
calculations, as a fracture diagnosis may have been missed in 
cases where radiographs were not obtained. Of the minority of 
patients who did return to the ED within 2 weeks with similar 
complaints, however, none had a missed fracture. These 
sensitivity and specificity values are consistent with results 
published in the literature. 7,9 

As noted in multiple reviews, a multi-faceted approach 
is often the most effective technique in enhancing adherence 
to clinical guidelines. 1718 As such, several interventions could 
be employed to improve Ottawa Knee Rule compliance. For 
example, reminders could be introduced by incorporating a 
diagram outlining the Ottawa Knee Rule criteria on ED history 
and physical examination templates. Alternatively, prompts 
can be integrated into electronic ordering systems asking the 
provider whether or not the patient has satisfied rule criteria 
when ordering a knee radiograph. Similar computerized 
decision support systems have yielded significant benefits on 
provider performance outcomes. 19 The results of our study 
also highlight the importance of focusing these educational 
and system level efforts not only on attending physicians but 
all ED providers. 

LIMITATIONS 

The physician survey and medical record review share 
certain common limitations. Our ability to generalize 
our findings is limited since the survey only queried ED 
attending physicians (it did not query ED nurses, residents, 
or physician assistants), and the medical records were from 
three hospitals affiliated with the same academic institution. 
Additionally, this study investigated an academic emergency 
medicine population that worked an average of 22.3 clinical 
hours per week, and consequently may not represent 
general community emergency physician practices. Specific 
limitations exist for the online survey. Our response rate of 
61.8%, while acceptable for physician surveys, is not ideal. 20 



Furthermore, while the questions assessing Ottawa Knee 
Rule knowledge were carefully designed by the authors, they 
were not validated or reviewed by a committee to establish 
face validity. In addition, the clinical vignettes involved 
patients aged 14-16 years old. Note that although the Ottawa 
Knee Rule has been validated in the pediatric population, the 
original study was conducted in patients aged 1 8 years or 
older. Additionally, using a more open-ended question when 
exploring barriers to implementation may have offered more 
diverse responses, although this would not have necessarily 
made for a stronger design. Moreover, it is possible that 
respondents gave socially-desirable answers to the survey that 
may not reflect their actual practice. This, however, is less 
likely considering that the survey was administered without 
being linked to the respondent's name. Several factors also 
limit our record review. Missing data from records limited 
our analysis. Furthermore, even in cases where all data was 
present, it is impossible to determine whether compliance 
was secondary to coincidence or conscious determination by 
healthcare providers to employ the decision tool. Nonetheless, 
this would have the effect of overestimating the adherence 
rate, suggesting that actual compliance is lower than our 
results indicate. 

CONCLUSION 

Compliance with the Ottawa Knee Rule among academic 
ED healthcare providers is poor. Patient concerns and system 
issues, rather than issues intrinsic to the rule itself, continue 
to serve as barriers to proper implementation of this validated 
decision tool. Addressing these concerns is essential to 
maximizing guideline adherence and mitigating unnecessary 
imaging. Improving compliance will require a comprehensive 
approach involving both education (of attending and non- 
attending providers alike) and system interventions, such as 
have been used with other clinical decision rules. 
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